

Overview 
of Findings 


The good 

• The new facet design greatly increased the visibility and usability of the 
filter area. Users in the Control Group took an average of 2 minutes and 
30 seconds to find the facet list. Users who saw the updated facet 
designs took 25 seconds on average to locate the filter area. 

• Both forms of filter logic were easy for users to get the hang of, 
although the majority of users expected the new (OR based) filter logic. 

• Users of the new design had an easier time 'reading' which filters they 
had previously applied to a collection page. 

The not so good 

• Collection and sub collection hierarchy was confusing for users. Users 
did not expect to see (sub)collections interspersed with items. 

• Users had trouble returning to the homepage and often looked for a 
link back on the far left of the navbar. 

• Users wanted more guidance in using Archive.org. Users were unsure of 
what types of content to look for and wanted to browse by genre. 



Overview 
of Findings 


High-level themes 

• The new filter design greatly decreased the amount of time that it took 
for users to locate collection facets and also made it easier for users to 
tell which facets were currently applied. All users of the new design said 
it was easy to use, fell within familiar or preferred design patterns and 
acted as they had expected. 

• The new OR based facet logic was slightly more predictable (and 
preferred) for first time users. However users were generally flexible and 
amenable to which ever facet logic was present in the version of the site 
they performed their session in. 

• First time users wanted more guidance into the Archive's collections. 
Users were quickly overwhelmed by the depth and breadth of the 
Archive's items and collections. Users wanted to be able to view 
content by entity, event (context) or genre and wanted better signaling 
of where 'high quality' or unique items in the Archive might be located. 

• The navbar and iconography were hard for many users to comprehend. 
Users often had trouble identifying the icons and expected - once 
clicked - for the icons to open to a new page or act as a filter. 



15 new users recruited from Usertesting.com performed 6 
tasks on the homepage and collections pages through an 
unmoderated and recorded session. 

The goals of the sessions were to: 

• Determine which (if either) method to approaching facet relationships is 
the most intuitive and easy for users to learn and use. 

• Are the filters easily recognizable and findable? 

• Can the user successfully utilize filters to locate relevant information? 

• Do the facets respond in a way that users expect? 

• What are users first impressions ofArchive.org? 

• How do users initiate a search for an item of personal interest to them 
on Archive.org? 


About 
the Test 



Users were broken up into three groups: 


AND Facet Logic 

Tested on Tracey's Dev Box 
Testing of new facet design 
with current facet logic 


Mixed OR Facet Logic 

Tested on Tracey's Dev Box 
Testing of new facet design 
with new facet logic 


Control Group 

Tested on the live site 
Testing of current facet design 
with current facet logic 


Dive-in to 
the Sessions 



Task 

Analysis 


Homepage 

• The 'About' blurb on the homepage was usually the first element that 
stood out to users. Users were attracted to wide variety of media types 
present on the Archive as well as the counts per media type. 

• The overall look and feel of the homepage and collection pages felt 
'drab,' 'dated,' or 'boring' to a number of users (particularly British 
users). These comments were particularly focused on the background 
and collection tile colors on the homepage. 

• Although users liked the images on the collection tiles, some users 
expressed that the tiles looked 'messy' or were hard to navigate. Users 
wanted the collections on the homepage to be organized either 
alphabetically or by subject matter and the infinite list of collections was 
overwhelming for some users. 

• The Wayback Machine search box in the navbar attracted a lot of 
attention from users but was confusing for most users. On first 
inspection some users thought it was a banner ad based on its 
placement and the design of the logo. 



Task 

Analysis 

Continued 


Collections Page 

• Collection hierarchy was confusing for users. The relationship between 
the collections on the homepage, subcollections, media type based 
collections and individual items was unclear for most users. 
Subcollections were generally read by users as links to a new page 
which would have more information about a single entity, event or 
genre along with relevant items. 

• The relationship between items in a collection was unclear on 
Community collections and the Etree collection (most users did not 
recognize any of the bands on the Etree collection and were therefore 
unsure what the collection was built around). Users wanted more 
information about a collection on the collection tile and at the top of 
the collection. Very few users navigated to the About tab. 

• Users had a hard time exiting the collection page. Those who wanted to 
see more items of the same mediatype appeared to be unsure of where 
to navigate to. Many users returned to the homepage to begin a new 
search. 



Facets 


Task 

Analysis 

Continued 


• Case-sensitive facets (Rock vs rock) were frustrating to users who 
wanted to see all of the items associated with a specific topic or 
subject. Even with the OR facet logic in place, users still had trouble 
ensuring that all of the relevant facets (upper, lower, mixed case) were 
selected. This lead to only a subset of all results being exposed. 

• Users quickly found the 'More' link, but were overwhelmed by the total 
number of facets. The original design for this page is a paginated 
model, something a users suggested during testing. Additionally, users 
expected this list to be in alphabetical order. 

• The new facet design greatly increased the visibility and usability of the 
filter area. Users in the Control Group took an average of 2 minutes and 
30 seconds to find the facet list. Users who saw the updated facet 
designs took 25 seconds on average to locate the filter area. 

• The new OR based facet logic was slightly more predictable (and 
preferred) for first time users. However users were generally flexible and 
amenable to which ever facet logic was present in the version of the site 
they performed their session in. 



Task 

Analysis 

Continued 


Free Explore Time 

• First time users wanted more guidance into the Archive's collections. 
Users were quickly overwhelmed by the depth and breadth of the 
Archive's items and collections. Users wanted to be able to view 
content by entity, event (context) or genre and wanted better signaling 
of where 'high quality' or unique items in the Archive might be located. 

• Many users began their free explore time by looking for a way to 
browse content within a specific media type. Users were quick to find 
the icons on the navigation bar, but were surprised that the icons did 
not lead directly to 'media specific' pages (eg. images.google.com or 
books.google.com) . Additionally some of the icons were confusing to 
users especially Audio and Software. 

• Community and media type collections were often overwhelming for 
users, who wanted to be able to browse the collection as a way to learn 
what types of items might be included in the collection. Commonly 
they looked for signals based on genre, similar to a user browsing on 
Netflix. 



Key Insights 
and Findings 
Continued 


Media Type Filtered Search 

Although users generally had a positive reaction to the wide variety of 
media types featured on the Internet Archive, users - especially during the 
free explore task - looked for ways to limit their browsing and searching by 
media type. These were users who had successfully utilized the collection 
facets and had noted the ability to filter by media type and yet they still 
looked for ways to narrow before beginning their search. 


Recommendation 


Q v Search 


Search the Entire 
Internet Archive 




Art / Images 
Collections 


L/al9 

eBooks 

Events 

Geography 

Music 

People 

Saci a MoHia 


JUUol ivitru la 

Television 

Websites 


1 1 

Advanced Search 


Enabling users to quickly 
narrow down their search 
from the search bar could 
help users feel confident 
and in control of their 
searches and reach relevant 
information faster. This filter 
area could also be used to 
surface advanced search 
throughout the website. 


Suggested ‘Search Filter’ Design 





Key Insights 
and Findings 
Continued 


Create a Home for Each Media Type 

Most users began looking for an item of interest to them by looking for a 
way to browse content within a specific media type. Users looked for a 
page that was specific to a single media type as a way to specify before 
exploring by genre, event or contributing institution. 

Recommendation 


Web Collection 

Search and browse 15 years of .atl sites 




Visual Showcasing of Media Type Pages on the Homepage 


Create a 'home' for each media type that is visually more similar to a 
homepage than a collection, including visually highlighted collections or 
genres as well as featured content. Link to these pages through a 
showcase on the homepage and through a redesign of the navigation bar. 








Key Insights 
and Findings 


Highlighting of Unique and High Quality Content 

The amount and breadth of the items available to users on the Archive was 
exciting but often overwhelming for the majority of users who were unsure 
of what specifically the was available through the Internet Archive. 


Recommendation 




Volta 

by BfOf* 


~ Atlantis CD Coll 
^ _ hlaliOftdJ Library 


Sugar Cubes in Japan 
Photos 

by Will Strands 

H University of Atlantis 



Dancer In The Darfc 
Behind the scenes 

by Lars Von Trier 


Univ 

Johr 


Paper Magazine Photo 
Shoots 



FEATURED ITEM 

Stonemilker Virtual Reality Video 

Pea horseradish azuki bean lettuce avocado asparagus okra. 
Kohlrabi radish okra azuki bean com fava bean mustard 
tigernut jkama green bean celtuce collard greens avocado 


Suggested ‘Featured Item’ Design 


Create a new Ul element that would enable 
Internet Archive staff, partners and 
collection creators to highlight and annotate 
featured items in a collection. 

These items could appear mixed in among 
other items in the collection and serve as a 
way to guide users to special items. The 
space for annotation would allow for 
contributors to share what makes this item 
noteworthy. These featured items could also 
be displayed on the homepage, as a way to 
browse across collections. 




Key Insights 
and Findings 
Continued 


Collection and Sub Collection Hierarchy 

The intermixing of collections and items on collection pages was 
disorienting for some users 

Recommendation 


Media Type 


20 COLLECTIONS 


texts 1524 

memos 632 

audo 323 

collections 20 


Dpics & Subjects 

Cnctai 
Yoga 
Cats 
Dance 
Astrology 
lore ► 


1843 

60S 

564 

548 

473 


flection | &» 

HoOoos and Hifinks 302 

A Brief Exp oration of 244 

Cat Naps 

The Cactus Cryptograph 235 

Trouble with Tabbies 1 86 

Contemplate the Catn p 1 43 


Hodooa and Hijinka 






- . u\n . 


3.932 ITEMS 


A Brief Exploration 
of Cat Naps 


The Cactus 
Crytcgraphy 


Trouble with Tabbies 


Contemplate the 
Catnip 






Title > Date > Creator 



IkhfiOT rfT Yngift: l 

by Of Mmu W#»t 


■ 5 GRID = LIST 

\m 

' £% - 

I Hi 


Suggested changes to the collection page 


In this design sub collections are separated from item results on the 
collection page. Additionally collection tile height is standardized as some 
users thought that larger collection tiles were a sign of a larger collection. 




Key Insights 
and Findings 
Continued 


Navigation Bar + Iconography 

Some users had trouble returning to the homepage once they navigated 
away and often looked for a link back on the far left of the navbar. More 
generally, the Top Hat' (clicking on a item opens above the navigation bar) 
was disorienting for some users, who expected the navigation bar to have a 
dropdown. Additionally, some users expected clicking on a media type icon 
to open to a new page or act as a filter over the current page. 

Legibility of icons may also be a problem as users relied primarily on 'helper 
text' to decode icons. 

Recommendation 

Our navigation bar is burdened by the quantity of content that needs to be 
accessible through it. Many other websites face similar challenges and 
looking at how they approach this design problem could help to influence 
further designs on the Internet Archive (please see the next slide for some 
examples). Higher visibility (expansion) of the search bar, a 'sticky' 
navigation bar, reorganization of submenus and increased text (over 
iconography) should be explored. 



Examples of High Content Navigation Bars 


Key Insights 
and Findings 
Continued 


Bloomberg the Company A Its Products 


Bloomberg Anywhere Remote Login | Bloomberg Terminal Demo Request 


Bloomberg 

Businessweek 


Tech 


Pursuits Politics 


Opinion 


Sign In 
Subscribe 


Stocks Currenc 

Regional Overview 
Stocks 

Stock Futures 
Gainers ♦ Losers 
Company Profiles 
Earnings Calendar 


Commodities Rates 4- Bonds Economics Magazine Benchmark Watchllst Economic Calendar 


Dow Jones Indus. Avg SAP 500 


FTSE100 


-0.60 % -0.46 o/o -0.43 


Nikkei 225 



New York 
Public 

Library 


LOG IN - 


LOCATIONS 


GET A LIBRARY CARD GET EMAIL UPDATES 




DONATE 


SHOP 




Browse 

Learn Attend Research Give Get Help 

Browse 


LOG IN 


RECOMMENDATIONS 

Books/Mustc/DVDs 

Recommendations 

4ft 

MyNYPL 


Staff Picks 

Blog 

Public Projects 
Shop 


Log into your account to check holds, item 
due dates, and more. 


There is a huge world of books out there. 
Our expert staff members pick out their 
favorites to help you find your next one. 


These two navigation bars from Bloomberg and NYPL, are content heavy 
but create clear pathways for users to access relevant information through 
color contrast, space between elements and clear navigation indicators. 
The Bloomberg navigation bar also includes a top bar connecting all 
Bloomberg web properties together through a single navigation system 







Additional Testing with Current Users 

Testing the two facet logics with current users of the Internet Archive 
would help to better understand the types of tasks that expert users of the 
site rely on collection and search facets for. The new users tested in this 
round of testing for the most part seemed amenable to either facet model, 
as both are models they come across while using the internet. It should be 
stated that the OR model seemed to be more expected by new users, 
however input from current users could help to determine which model is 
a better fit for Archive.org. 

A/B Testing of Both Facet Logics 

An alternative to more formal testing with current users, or incase current 
users are still split between AND and OR facet logic, live A/B testing of 
both facet logics would be helpful. Success metrics would need to be 
defined in order to determine which logic was the most successful. 


Next Steps 
for Collections 



Questions, comments, or concerns? 

Please don't hesitate to reach out to me on Slack, 
in person or by email at carolyn@archive.org 



